{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Lab: Cost Function for Logistic Regression\n",
    "\n",
    "## Goals\n",
    "In this lab, you will:\n",
    "- examine the implementation and utilize the cost function for logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "%matplotlib widget\n",
    "import matplotlib.pyplot as plt\n",
    "from lab_utils_common import  plot_data, sigmoid, dlc\n",
    "plt.style.use('./deeplearning.mplstyle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset \n",
    "Let's start with the same dataset as was used in the decision boundary lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])  #(m,n)\n",
    "y_train = np.array([0, 0, 0, 1, 1, 1])                                           #(m,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a helper function to plot this data. The data points with label $y=1$ are shown as red crosses, while the data points with label $y=0$ are shown as blue circles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig,ax = plt.subplots(1,1,figsize=(4,4))\n",
    "plot_data(X_train, y_train, ax)\n",
    "\n",
    "# Set both axes to be from 0-4\n",
    "ax.axis([0, 4, 0, 3.5])\n",
    "ax.set_ylabel('$x_1$', fontsize=12)\n",
    "ax.set_xlabel('$x_0$', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost function\n",
    "\n",
    "In a previous lab, you developed the *logistic loss* function. Recall, loss is defined to apply to one example. Here you combine the losses to form the **cost**, which includes all the examples.\n",
    "\n",
    "\n",
    "Recall that for logistic regression, the cost function is of the form \n",
    "\n",
    "$$ J(\\mathbf{w},b) = \\frac{1}{m} \\sum_{i=0}^{m-1} \\left[ loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)}) \\right] \\tag{1}$$\n",
    "\n",
    "where\n",
    "* $loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)})$ is the cost for a single data point, which is:\n",
    "\n",
    "    $$loss(f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}), y^{(i)}) = -y^{(i)} \\log\\left(f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) - \\left( 1 - y^{(i)}\\right) \\log \\left( 1 - f_{\\mathbf{w},b}\\left( \\mathbf{x}^{(i)} \\right) \\right) \\tag{2}$$\n",
    "    \n",
    "*  where m is the number of training examples in the data set and:\n",
    "$$\n",
    "\\begin{align}\n",
    "  f_{\\mathbf{w},b}(\\mathbf{x^{(i)}}) &= g(z^{(i)})\\tag{3} \\\\\n",
    "  z^{(i)} &= \\mathbf{w} \\cdot \\mathbf{x}^{(i)}+ b\\tag{4} \\\\\n",
    "  g(z^{(i)}) &= \\frac{1}{1+e^{-z^{(i)}}}\\tag{5} \n",
    "\\end{align}\n",
    "$$\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-02'></a>\n",
    "#### Code Description\n",
    "\n",
    "The algorithm for `compute_cost_logistic` loops over all the examples calculating the loss for each example summing.\n",
    "\n",
    "Note that the variables X and y are not scalar values but matrices of shape ($m, n$) and ($𝑚$,) respectively, where  $𝑛$ is the number of features and $𝑚$ is the number of training examples.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost_logistic(X, y, w, b):\n",
    "    \"\"\"\n",
    "    Computes cost\n",
    "\n",
    "    Args:\n",
    "      X (ndarray (m,n)): Data, m examples with n features\n",
    "      y (ndarray (m,)) : target values\n",
    "      w (ndarray (n,)) : model parameters  \n",
    "      b (scalar)       : model parameter\n",
    "      \n",
    "    Returns:\n",
    "      cost (scalar): cost\n",
    "    \"\"\"\n",
    "\n",
    "    m = X.shape[0]\n",
    "    cost = 0.0\n",
    "    for i in range(m):\n",
    "        z_i = np.dot(X[i],w) + b\n",
    "        f_wb_i = sigmoid(z_i)\n",
    "        cost +=  -y[i]*np.log(f_wb_i) - (1-y[i])*np.log(1-f_wb_i)\n",
    "             \n",
    "    cost = cost / m\n",
    "    return cost\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the implementation of the cost function using the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w_tmp = np.array([1,1])\n",
    "b_tmp = -3\n",
    "print(compute_cost_logistic(X_train, y_train, w_tmp, b_tmp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**: 0.3668667864055175"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "Now, let's see what the cost function output is for a different value of $w$. \n",
    "\n",
    "* In a previous lab, you plotted the decision boundary for  $b = -3, w_0 = 1, w_1 = 1$. That is, you had `w = np.array([-3,1,1])`.\n",
    "\n",
    "* Let's say you want to see if $b = -4, w_0 = 1, w_1 = 1$, or `w = np.array([-4,1,1])` provides a better model.\n",
    "\n",
    "Let's first plot the decision boundary for these two different $b$ values to see which one fits the data better.\n",
    "\n",
    "* For $b = -3, w_0 = 1, w_1 = 1$, we'll plot $-3 + x_0+x_1 = 0$ (shown in blue)\n",
    "* For $b = -4, w_0 = 1, w_1 = 1$, we'll plot $-4 + x_0+x_1 = 0$ (shown in magenta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Choose values between 0 and 6\n",
    "x0 = np.arange(0,6)\n",
    "\n",
    "# Plot the two decision boundaries\n",
    "x1 = 3 - x0\n",
    "x1_other = 4 - x0\n",
    "\n",
    "fig,ax = plt.subplots(1, 1, figsize=(4,4))\n",
    "# Plot the decision boundary\n",
    "ax.plot(x0,x1, c=dlc[\"dlblue\"], label=\"$b$=-3\")\n",
    "ax.plot(x0,x1_other, c=dlc[\"dlmagenta\"], label=\"$b$=-4\")\n",
    "ax.axis([0, 4, 0, 4])\n",
    "\n",
    "# Plot the original data\n",
    "plot_data(X_train,y_train,ax)\n",
    "ax.axis([0, 4, 0, 4])\n",
    "ax.set_ylabel('$x_1$', fontsize=12)\n",
    "ax.set_xlabel('$x_0$', fontsize=12)\n",
    "plt.legend(loc=\"upper right\")\n",
    "plt.title(\"Decision Boundary\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see from this plot that `w = np.array([-4,1,1])` is a worse model for the training data. Let's see if the cost function implementation reflects this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w_array1 = np.array([1,1])\n",
    "b_1 = -3\n",
    "w_array2 = np.array([1,1])\n",
    "b_2 = -4\n",
    "\n",
    "print(\"Cost for b = -3 : \", compute_cost_logistic(X_train, y_train, w_array1, b_1))\n",
    "print(\"Cost for b = -4 : \", compute_cost_logistic(X_train, y_train, w_array2, b_2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**\n",
    "\n",
    "Cost for b = -3 :  0.3668667864055175\n",
    "\n",
    "Cost for b = -4 :  0.5036808636748461\n",
    "\n",
    "\n",
    "You can see the cost function behaves as expected and the cost for `w = np.array([-4,1,1])` is indeed higher than the cost for `w = np.array([-3,1,1])`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "In this lab you examined and utilized the cost function for logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
